guilt by association

n : the attribution of guilt (without proof) to individuals

because the people they associate with are guilty